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Abstract: This study investigated thefactors that shape the broader impacts activities ofNSF 
grant recipients. A random sample ofNSFgrantees was surveyed about the type and quality of 
their broader impacts activities, their views on knowledgeproduction and the democratization 
of science, their experience and training, and the existence of a supportive climate and 
resources for community engagement at their home institutions. Respondents indicated that 
they shared an orientation towards knowledge production that was more detnocratic than 
technocratic and valued public engagement in science; that they had adequate experience 
but little training in community engaged activities and lacked confidence in their ability 
to evaluate such work; that executive leaders at their institutions encouraged community 
engagement but promotion and tenure policies did not recognize such activities; and that 
they had little access to training, funding, or infrastructure to support community engaged 
activities. A multinomial logistic regression showed that faculty expertise, available resources, 
and academic discipline were the strongest predictors of type of broader impacts activity 
(p < .001). A multiple regression analysis revealed that faculty expertise, a democratic 
orientation towards knowledge production, and a supportive climate were the strongest 
predictors of quality of broader impacts activity (p < . 001). 
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Background and Objectives 

An enduring image of academia is that of an ivory tower, disconnected from the messy problems of 
the world. It is a recurring complaint that the primacy of basic research and the academy’s emphasis 
on abstract theory has eroded higher education’s connection to the world, isolating scholars from 
society and making their work obscure or irrelevant to the general public (Barker & Brown, 2009; 
Gibson, 2006). Decreasing appropriations for higher education heighten the need to convince 
the public about the value of university research. In response to prodding from Congress, federal 
funding agencies are increasingly requiring academic research grant proposals to include indicators 
of public impact that would result in social good. 

In 1997, the National Science Foundation (NSF) established a policy that all funding proposals 
submitted to the agency would be evaluated on two criteria: intellectual merit and broader 
impacts. Broader impacts refer to “specific, desired societal outcomes” (NSF, 2012, p. III-2) such as 
the participation of underrepresented groups in science, technology, engineering, and mathematics 
(STEM); enhancing STEM education; public scientific literacy and engagement; and partnerships 
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between academia, industry, and others. According to Arden Bement, former director of the NSF, 
“The [broader impacts] criterion was established to get scientists out of their ivory towers and 
connect them to society” (quoted in Lok, 2010, p. 416). 

A national review of the effectiveness of these criteria conducted in 2000 revealed that, while most 
researchers had little difficulty identifying the intellectual merit of their plans, many struggled 
to adequately articulate the broader impacts of their proposed investigations (NSF, 2005). The 
study also found that the broader impacts criterion was consistently weighted less than intellectual 
merit in the proposal review process and that many in the scientific community were resistant 
to or dismissive of the requirement (NSF, 2005). The NSF responded with efforts to educate 
the scientific community about the agency’s rationale and expectations for broader impacts. 
A subsequent national evaluation of the NSF’s review criteria conducted in 2010 found that 
problems with the execution, understanding, and acceptance of the broader impacts criterion 
persisted, that assessment was unclear and inconsistent, that there was little variety in the type of 
activities performed to address the broader impacts criterion, and that principal investigators (Pis) 
needed greater institutional support to respond effectively to this requirement (NSB,201l). 

The purpose of this study was to investigate the factors that shape PI response to the NSF broader 
impacts criterion. The following research questions guided the study: 

1. What are the different types of activity that Pis engage in to meet the broader impacts 
criterion of NSF-sponsored research? 

2. What is the quality of the broader impacts activities, as assessed by Pis using evaluation 
criteria for the scholarship of engagement? 

3. To what extent do Pis’ individual characteristics (expertise, epistemology, academic 
discipline, and rank) and institutional support (climate and resources) for community 
engagement predict the type of their broader impacts activities? 

4. To what extent do Pis’ individual characteristics and institutional support predict the 
quality of their broader impacts activities? 

The author gathered survey data to identify the types of broader impacts activities that Pis 
conduct, and their perceptions of the quality of these activities. The author then determined how 
much variance in type and quality of activity could be explained by the respondents’ personal 
characteristics and perceptions of institutional support for community engagement. In the 
survey, community engagement was defined as “The collaboration between institutions of higher 
education and their larger communities (local, regional/state, national, global) for the mutually 
beneficial exchange of knowledge and resources in a context of partnership and reciprocity” 
(Carnegie Foundation for the Advancement of Teaching, n.d., n.p.). 

Conceptual Framework 

This study is grounded in the theoretical framework of the scholarship of engagement. Such 
scholarship is academically relevant faculty work that simultaneously meets campus goals and 
community needs, incorporates community issues, and is integrative across teaching, research, 
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and service (Clearinghouse and National Review Board for the Scholarship of Engagement, 
2002). Although there are subtle distinctions between the terms “scholarship of engagement,” 
“engaged scholarship,” and “community-engaged research,” they are used interchangeably 
throughout this study. 

The NSFs broader impacts merit review criterion has been an ongoing challenge within the 
scientific community, particularly in terms of conceptual clarity, assessment, and philosophical 
resistance. The persistence of the problem may stem, in part, from the paucity of literature on the 
broader impacts of academic research (Buxton, 2011). While scant research has been conducted 
on such impacts, volumes have been written on the scholarship of engagement. The engagement 
movement has spawned multiple journals, institutes, conferences, and consortia dedicated to 
studying engaged scholarship in its many forms. This study drew from this extensive literature 
base to better understand the issues that surround the broader impacts of academic research 
and connections between the academy and society. Framing the study within the context of the 
scholarship of engagement is appropriate not only because broader impacts activities may be viewed 
as an expression of engaged scholarship, but also because both struggle to achieve legitimacy and 
support within the academy. 

The Scholarship of Engagement 

The concept of engaged scholarship is often credited to Ernest Boyer, a renowned leader of 
educational reform. Boyer (1990) argued that the traditional definition of scholarship as research 
that advances disciplinary frontiers of knowledge was too limited. He countered the prevailing 
hierarchical view of scholarship with a more inclusive vision that added to the ranks of conducting 
original research such activities as identifying connections between concepts, bridging theory 
and practice, and effectively communicating knowledge. Boyer posited that faculty scholarly 
work included four “separate, yet overlapping, functions” (1990, p. 16) which he identified as the 
scholarships of discovery, integration, application, and teaching. Boyer (1996) proposed the term 
scholarship of engagement as interaction across these four realms to address community needs. He 
described the scholarship of engagement as “connecting the rich resources of the university to the 
most pressing social, civic, and ethical problems...creating a special climate in which the academic 
and civic cultures communicate more continuously and more creatively with each other” (pp. 19- 
20). Boyer asserted that it was time for higher education to renew its covenant with society and to 
partner with communities in directly addressing problems, with a reciprocal flow of knowledge. 

Models of Knowledge Production 

Epistemologies about knowledge production and distribution shape perceptions about the role 
of higher education in society, define relationships between campus and community, and govern 
the values, norms, and practices of scholars. Boyer raised critical questions of how knowledge is 
constructed and what is accepted as legitimate knowledge in the academy. Deeply embedded in 
academia is the traditional linear model of knowledge production and distribution, which prizes 
basic research that is disciplinary, removed from influence by outside interests, and conducted 
without consideration of use (Bush, 1945). Boyer countered the prevailing belief that basic 
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research was the most essential form of scholarly activity, with publications and teaching flowing 
from it. He argued that “knowledge is not necessarily developed in such a linear manner. The arrow 
of causality can, and frequently does, point in both directions. Theory surely leads to practice. 
But practice also leads to theory” (1990, p. 16). The national dialogue that ensued prompted 
academics to identify new models of knowledge creation, including that of use-inspired research 
(Stokes, 1997) and Mode 2 science (Gibbons et ah, 1994). 

Linear 

The dominant, linear model ofknowledge production begins with basic research, which may lead 
to applied research and technological development, and on to production or operations. This 
model has shaped U.S. science policy since it was first proposed by Vannevar Bush in his 1945 
report to President Roosevelt (Mazuzan, 1994). Bush saw an inherent tension between the goals 
of understanding and use, and warned that the creativity of basic research would be stifled by 
premature thoughts of practical use. 

Use-inspired 

One year after Boyers call for engaged scholarship, Stokes challenged the traditional dichotomy 
between basic and applied research. Stokes (1997) argued that Bushs premise that flows between 
science and technology are uniformly one way, from scientific discovery to technological innovation, 
was flawed and offered a limited understanding of how knowledge is generated and put to use. 
Stokes contested Bushs cannon that the goals of understanding and use are dichotomous, arguing 
that research is often influenced by both goals. In his model, theoretical and practical research and 
application come together to create a dynamic cycle of innovation driven by changing conditions 
and the competitive landscape. 

Mode 2 

Gibbons and his co-authors (1994) offered another alternative to the static linear model of 
knowledge production. They determined that Bush’s old paradigm of discipline-specific, 
autonomous, university-grounded scientific discovery (“Mode 1”) was being supplanted by a new 
approach to knowledge production (“Mode 2”). Gibbons et al. posited that the new mode emerged 
from a parallel expansion in the demand for specialist knowledge and the number of potential 
knowledge producers, adding government laboratories, industry, consultancies, and others to 
the domain formerly ruled by universities. They described this new mode as socially distributed, 
application-oriented, trans-disciplinary, and subject to multiple accountabilities. This evolution, 
they argued, occurred in response to significant trends changing the research environment: the 
steering of research priorities, the commercialization of research, and the accountability of science. 
As a result, the research process was no longer an objective investigation, but rather an intense 
dialogue between research actors and subjects. 

Reciprocal 

In the dominant, technocratic culture of higher education, knowledge flows in one direction: from 
credentialed, detached experts ensconced in the university to its place of need and application in 
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the community (Hartley, Saltmarsh, & Clayton, 2010). The scholarship of engagement challenges 
the belief that knowledge is produced by academics and then transferred to the community. Instead, 
engaged scholarship builds reciprocal relationships between scholars and community partners, 
dynamic synergies between theory and practice, and research that benefits both the community 
and the academy (Nicoreta, Cutforth, Fretz, & Thompson, 2011). Such reciprocity has been 
defined as “democratic engagement” and described as “an epistemological shift...that favors mutual 
deference between lay persons and academics” (Hartley, Saltmarsh, & Clayton, 2010, p. 401). 

Study Model 

The study’s conceptual model, depicted in Figure 1, illustrates the congruence of NSF goals and 
broader impacts activities (bulleted lists) with Boyers domains of scholarship and the models of 
knowledge production described above. The model aligns these elements along the oppositional 
axes of democratic versus technocratic orientations and the goals of improving understanding 
versus improving technology. 



Figure 1. Alignment between NSF goals, Boyer’s concept of engaged scholarship, 
and models of knowledge production. 


Within the context of this conceptual framework, the author explored the relationships between 
the individual characteristics of faculty members, support at their home institutions, and the type 
and quality of their broader impacts activities. The scholarship of engagement literature suggests 
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possible associations between engagement in public scholarship and individual characteristics such 
as epistemology (Austin & Beck, 2010) and expertise (Baskurt, 2011), as well as demographic 
factors including discipline (Lunsford & Omae, 2011) and rank (Glass, Doberneck, & Schweitzer, 
2011). The literature also indicates that institutions influence individual behavior by means of 
overall climate (Vogelgesang, Denson, & Jayakumar, 2010) and the allocation of resources for 
administrative support, faculty development, and incentives (Doberneck, Brown, & Allen, 2010). 
Figure 2 outlines the variables investigated in this study. 



Figure 2. Variables that shape broader impacts activities of academic faculty. 


Tire Journal of Research Administration, (47)1 











































































74 Nagy 


Methods 

Population 

The population for this study was the 3,635 faculty members at institutions of higher education 
across the United States who received NSF research grants during Fiscal Tear 2009 (October 1, 
2008 - September 30, 2009). This period was selected because it not only follows on the heels 
of the NSF s 2007 release of broader impacts criterion guidelines, but it also allows sufficient 
time for the completion of most projects. Research grant is a term used by the NSF to represent 
what may be considered a typical research award, particularly with respect to the award size (NSF, 
2009). Education research grants are included in this category. Excluded are large awards such 
as centers and facilities, as are equipment and instrumentation grants. Also excluded are grants 
for conferences and symposia, grants in the Small Business Innovation Research program, Small 
Grants for Exploratory Research, and education and training grants. EAGER and RAPID grants 
were also excluded as they are exempt from external review. 

Sample 

A stratified random sample of 700 grant recipients was drawn from the population. This sample 
size was chosen to provide a pool of respondents large enough to restrict error to 5%, assuming a 
response rate of approximately 50% (Patten, 2002). A stratified random sample design was selected 
to reduce normal sample variation and to produce a sample more likely to resemble the total 
population than a simple random sample. Given the importance attributed to academic discipline 
in the literature, the sample was stratified along NSF Directorates to provide representation across 
a broad spectrum of disciplines. One hundred grant recipients were selected from each of the 
following directorates: Biological Sciences; Computer and Information Science and Engineering; 
Education and Human Resources; Engineering; Geosciences; Mathematics and Physical Sciences; 
and Social, Behavioral, and Economic Sciences. The population of grant recipients in each 
directorate was assigned consecutive numbers and the desired sample was drawn using a table of 
random numbers. 

Instrument 

As no instrument existed to collect the data necessary for this study, the author developed a special 
purpose survey derived from an extensive review of the literature. The survey items generated 
fell into five subsections: broader impacts activities, epistemology, expertise, institutional 
support, and demographics. The first section asked participants about the type and quality of 
their broader impacts activities and community partnerships. The second section attempted to 
capture participants’ epistemologies by inquiring about their views on knowledge production 
and the democratization of science. The section on expertise asked participants about their prior 
experience, training, and sense of ability with activities similar to those conducted to address the 
broader impacts criterion, both as graduate students and as faculty members. The next section asked 
participants about the existence of a supportive climate and resources for community engagement 
at their home institutions. The final section of the survey gathered demographic information on 
academic discipline and rank. The questions on discipline, rank, and activity type were multiple 
choice; the remaining items used a five-point scale, ranging from 1 (not at alt) to 5 (fully ). 
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The next step in survey development was a pre-survey evaluation conducted by a panel of field 
experts in order to identify and correct any foreseeable problems prior to executing the survey. 
The panel members were selected based on their expertise in engaged scholarship, survey design, 
and sponsored research. The panel provided input on the conceptual accuracy of the items, the 
length and legibility of the scale, and any missing or misrepresented dimensions of the constructs 
to be studied. After integrating feedback from the expert panel into the instrument, the survey 
received an exempt review and approval from the Institutional Review Board of the University 
of South Dakota. 

Data Collection 

This cross-sectional study used an online survey to collect data from faculty at academic institutions 
across the United States. The primary concern about web-based surveys is that they may introduce 
error if online use is not universal. This concern does not apply to the study at hand because the 
sample population, university faculty, typically demonstrate competency in the use of email and 
other forms of online communication (Shannon, Johnson, Searcy, & Lott, 2002). Consequently, 
given considerations of convenience and economy, a web-based survey was deemed the most 
expedient approach to data collection. 

The survey was formatted online using the survey software QuestionPro. The researcher collected 
email addresses for members of the sample from the NSF website and sent each member an email 
message explaining the project, inviting their participation, and directing them to the survey 
web link. The survey was distributed on February 4, 2013, and remained open through March 1, 
2013. Follow-up with non-respondents consisted of reminder emails. All sample non-respondents 
were sent a first, second, and final email reminder after weeks one, two, and three, respectively. 
QuestionPro’s Respondent Anonymity Assurance was enabled on the survey to ensure that, 
although computer generated identification numbers for individuals were created, the researcher 
did not have simultaneous access to both a respondent’s email address and response data. 

Data Analysis 

The Statistical Package for the Social Sciences was used for all data analyses and the .05 level of 
significance was used for all inferential statistics. 

Research question 1. What are the different types of activity that Pis engage in to meet 
the broader impacts criterion of NSF-sponsored research? Type of broader impacts activity 
was a categorical variable. The categories, identified by the NSF, were education, broadening 
participation, infrastructure, dissemination, and other. Frequencies of responses to the question 
about type of activity were tabulated to derive percentages ofparticipation in each activity. 

Research question 2. What is the quality of the broader impacts activities, as assessed by 
Pis using evaluation criteria for the scholarship of engagement? fpuality of broader impacts 
activity was a summed score of responses to the questions about each area of assessment (goals, 
preparation, methods, results, presentation, critique, and ethics of the activity, as well as the 
reciprocity, communication, structures, and duration of community partnerships). Means 
and standard deviations generated from these items, and composite means for each element, 
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provided information related to the areas of relative strength and weakness of broader 
impacts activities. 

Research question 3. To what extent do Pis’ individual characteristics (expertise, 
epistemology, academic discipline, and rank) and institutional support (climate and resources) 
for community engagement predict the type of their broader impacts activities ? Expertise was 
a summed score of questions about the respondents prior experience, training, and sense 
of ability related to community-engaged activities. Epistemology was a summed score of 
questions about the respondent’s views on knowledge production and the democratization 
of science. Responses were grouped into two categories: democratic and technocratic 
orientations. Climate was a summed score of questions about the respondent’s perceptions of 
institutional climate as expressed through rhetoric, recognition, and rewards for community 
engagement. Resources was a summed score of questions about the respondent s perceptions 
of the administrative support, faculty development, and incentives available for community 
engagement. Academic discipline was operationalized as the NSF directorate to which the 
respondent applied. Academic rank was captured with dummy variables for assistant professor, 
associate professor, foil professor, research scientist, and other. 

After reverse coding negatively worded items and removing cases with missing observations 
on multiple indicators, a multiple logistic regression was conducted to determine how the 
variables ranked in terms of highest to lowest when it came to predicting the type of broader 
impacts activity. 

Research question 4. To what extent do Pis’ individual characteristics and institutional 
support predict the quality of their broader impacts activities? Quality of activity was a 
summed score of assessment questions based on the scholarship of engagement criteria. The 
independent variables were the constructs identified in the analysis of research question 3 
(individual characteristics and institutional support). Because the dependent variable was 
interval-scaled, a multiple regression analysis was employed. A stepwise multiple regression 
determined how much of the variance in quality of broader impacts activity scores could be 
explained by the respondents’ expertise, epistemology, rank, discipline, institutional climate, 
and resources. 

Results 

Response Rate 

The rate of response to the study’s survey was low. Of the 700 surveys distributed, 15 were mail 
returns, 246 were viewed, 209 were started, and 124 were completed. The completed response 
rate was 18.1%. Although the limited number of cases restricted the options for statistical analysis, 
the low response rate did not necessarily invalidate the survey. An examination of the results of 
81 national surveys with response rates ranging from 5% to 54% revealed that surveys with much 
lower response rates were only minimally less accurate (Holbrook, Krosnick, & Pfent, 2007). 
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Demographics 

Of the 124 respondents who completed the survey, half held the academic rank of full professor 
in = 62, 50.0%), while the bulk of the rest were associate professors (» = 40, 32.3%), with some 
assistant professors [n = 14, 11.3%). Table 1 reports the frequencies and percentages of the 
academic rank categories. 

Table 1. Frequencies and Percentages of Academic Rank 


Academic Rank 

n 

% 

Full professor 

62 

50.4 

Associate professor 

40 

32.5 

Assistant professor 

14 

11.4 

Research scientist/professor 

5 

4.1 

Other 

2 

1.6 


Note: n = 123; one response was missing. 


The degree to which the sample is representative of the population of NSF grantees is unknown 
because the NSF does not track the academic rank of their grantees. The NSF does, however, 
categorize their grantees as either “Early Career Pis” (investigators within seven years of receiving 
their last degree) or “Later Career Pis” (those who received their last degree more than seven years 
before the award date). In FY 2009, 78% of NSF grantees were classified as Later Career Pis; 22% 
were Early Career Pis (NSB, 2010). The fact that the promotion from Assistant to Associate 
Professor generally occurs within six years on the tenure track suggests that the distribution of the 
sample is reasonably representative of the population. 

The respondents were fairly evenly distributed across the seven NSF directorates, as Table 2 depicts. 
The Directorate of Biological Sciences had the largest representation (n = 27,21.8%); the next five 
directorates were closely clustered, with the number of respondents ranging from 19(15.3%) to 15 
(12.1%). The Directorate of Geosciences had the fewest respondents (n = 11, 8.9%). 

Table 2. Frequencies and Percentages of NSF Directorates 


NSF Directorate 

n 

% 

Biological sciences 

27 

21.8 

Mathematics and physical sciences 

19 

15.3 

Social behavioral and economic sciences 

18 

14.5 

Computer science and engineering 

16 

12.9 

Engineering 

16 

12.9 

Education and human resources 

15 

12.1 

Geosciences 

11 

8.9 


Note: n = 122; two responses were missing. 
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Type of Broader Impacts Activity 

Research question 1 asked about the different types of activity that Pis engage in. The categories 
of activity, identified by the NSF, were education, broadening participation, infrastructure, 
dissemination, and other (NSF, 2008a). Frequencies of responses to the question about type of 
activity were tabulated to derive percentages of participation in each activity. Education was by 
far the most common activity (w = 71, 57.3%), while dissemination was the second most reported 
activity (n = 29, 23.4%). The remaining categories were substantially lower. Descriptive statistics 
generated from this item are presented in Table 3. 

Table 3. Frequencies and Percentages of Type of Broader Impacts Activity 


Type 

n 

% 

Education 

62 

57.3 

Dissemination 

29 

23.4 

Broadening participation 

12 

9.7 

Infrastructure 

9 

7.3 

Other 

3 

2.4 


Quality of Broader Impacts Activities 

Respondents assessed the quality of their broader impacts activities in terms of goals, preparation, 
methods, results, presentation, critique, and ethics of the activity. The response scale was 1 {not at 
alt) to 5 fully). Respondents expressed the greatest confidence in their selection and use of methods 
(composite M = 4.18, SD = 0.64). High scoring items included “I used methods appropriate to 
my goals” {M = 4.36, SD = 0.65) and “I effectively applied the methods selected” {M = 4.11, 
SD = 0.79). Scores were also relatively high for preparation (composite M = 4.16, SD = 0.82), 
with the leading item being “I brought together the skills and resources necessary to move the 
project forward” {M =4.30, SD = 0.74). Interestingly, the area of ethics included both the highest 
and lowest scoring items: “I followed all policies concerning the responsible conduct of research” 
{M = 4.61, SD = 0.74) and “I subjected my activity to university IRB review” {M = 3.10, 
SD = 1.82). 

The area that showed the greatest weakness was critique (composite M = 3.49, SD = 0.98). The 
lowest scoring items in this category were “I used an appropriate breadth of evidence for the 
critique” {M =3.39, SD =1.11) and “I used this evaluation to improve the quality of future work” 
{M = 3.51, SD = 1.01). Respondents also rated themselves lower in using the results (composite 
M = 3.87, SD = 0.80). The items “I presented the results to both academic and non-academic 
audiences” and “The results of the activity added consequently to the field” received the relatively 
low mean scores of 3.73 {SD = 1.11) and 3.76 {SD = 0.97), respectively. 

Thirty-eight respondents (30.6%) reported that their broader impacts activities involved 
collaboration with a specific community agency (such as a school, museum, organization, or 
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other group). These respondents assessed their community partnerships in terms of reciprocity, 
communication, structures, and duration of the collaboration. The scoring for this area of 
assessment was noticeably lower than the others (composite M = 3.07, SD = 1.07). The highest 
scoring items were “Communications between the community partner and I were ongoing and 
bidirectional” {M = 3.53, SD = 1.13) and “The relationship continued after the project ended” 
{M = 3.47, SD = 1.43). The items receiving the lowest ratings were “A formal agreement outlined 
roles and responsibilities” {M = 2.55, SD = 1.33) and “The community partner and I shared 
power and responsibility equitably” {M = 2.74, SD = 1.25). 

Predictors 

The literature suggests possible associations between engagement in public scholarship and 
personal and institutional characteristics. This study investigated these variables as predictors of 
type and quality of broader impacts activities. 

Expertise. This variable was a summed score of questions about the respondents’ prior 
experience, training, and sense of ability related to community-engaged activities. Respondents 
indicated that they had a good deal of prior experience with activities similar to their broader 
impacts work as faculty members {M = 4.01, SD = 1.01), but they had received little faculty 
training for such work {M = 2.72, SD = 1.22). As graduate students, they received even less 
training {M = 2.39, SD = 1.35) and had less experience {M = 2.78, SD = 1.39). Respondents 
reported that they had a relatively strong sense of ability to implement broader impacts 
activities {M = 3.75, SD = 0.90), but they were less confident in their ability to evaluate these 
activities {M = 3.27, SD = 1.09). 

Epistemology. This variable was a summed score of questions about the respondents’ 
views on knowledge production and the democratization of science. Responses were 
grouped into two categories: democratic and technocratic orientations. A technocratic 
orientation is characterized by the valuing of basic, discipline-specific, curiosity-driven 
research conducted exclusively by highly specialized academic experts. A democratic 
orientation values applied, trans-disciplinary, problem-centered, use-inspired research that 
welcomes collaboration with partners outside the academy. Survey responses indicated 
orientations that were more democratic {M = 3.22, SD = 0.68) than technocratic 
{M = 2.85, SD = 0.68). Respondents tended to view public engagement in science as a 
way to increase acceptance of the results and generate information that meets user needs 
{M = 3.85, SD = 0.93), and to take into account public values and contexts as well as local 
expertise {M = 3.09, SD = 1.13). Respondents did not agree that public ignorance should 
be corrected through a one-way flow of information from the academy to the community 
{M = 2.17, SD = 1.01), or that scientific research would be most productive if left to scientists 
alone {M = 2.49, SD = 1.21). 

Climate. This variable was a summed score of questions about the respondents’ perceptions of 
institutional climate as expressed through rhetoric, recognition, and rewards for community 
engagement. The overall mean for this variable was moderate, at 3.02 {SD- 0.79). Respondents 
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indicated that the executive leadership at their institutions explicitly promoted community 
engagement as a priority ( M = 3.42, SD = 1.11), but their promotion and tenure policies did 
not reward such research (M = 2.15, SD = 0.93). 

Resources. This variable was a summed score of questions about the respondents’ 
perceptions of the administrative support, faculty development, and incentives available 
for community engagement. The overall mean for this item was low, at 2.31 (SD = 0.82). 
Respondents reported that faculty training was not available in engaged scholarship 
(M = 2.19, SD = 1.04) and that there was little financial support (M = 2.20, SD = 1.10) or 
infrastructure to facilitate collaboration (M - 2.38, SD = 1.07) and assessment (M = 2.40, 
SD = 1.06) of community engaged activities. 

Influence on Type of Activity 

A multinomial logistic regression analysis was conducted to predict the type of broader impacts 
activity the respondents engaged in, using expertise, epistemology, institutional support, discipline, 
and rank as predictors. A test of the full model against a constant only model was significant, 
indicating that the predictors, as a set, reliably distinguished between types of broader impacts 
activity, J( 2 (52, n = 102) = 87.61,/) = .001. A Nagelkerke’s R 2 of .644 indicated a moderately 
strong relationship between prediction and grouping. The model’s overall prediction success was 
68.3%. The likelihood ratio tests demonstrated that expertise, resources, and discipline contributed 
significantly to the model; expertise emerged as the strongest predictor (based on p value). Table 4 
presents the contribution of each variable to the model. 

Table 4. Likelihood Ratio Test of Each Predictor’s Contribution to the Model 


Predictor 

X 2 

df 

P 

Expertise 

22.61 

4 

.000* 

Resources 

17.28 

4 

.002* 

Discipline 

40.67 

24 

.018* 

Rank 

14.11 

8 

.079 

Technocratic 

5.13 

4 

.274 

Democratic 

0.67 

4 

.956 

Climate 

1.30 

4 

.861 


* Significant predictors at. 05. 


Influence on Quality of Activity 

A multiple regression analysis revealed that the linear combination of individual characteristics 
and institutional support was significantly related to quality, F( 15,84) = 5.16,/) = .000. The sample 
multiple correlation coefficient (R 2 ) was .70, indicating that approximately 49% of the variance 
in quality could be accounted for by the linear combination of individual characteristics and 
institutional support. A stepwise multiple regression presented a more parsimonious model with 
an equally significant regression equation, A(5,90) = 14.89 , p = .000, and an adjusted R 2 of .42. 
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The stepwise model revealed that expertise accounted for 20% of the variance, and that the 
inclusion of climate into the model resulted in an additional 8% of the variance being explained. 
Table 5 presents indices of the relative strength of each predictor. These results suggest that faculty 
expertise is the greatest predictor of the quality ofbroader impacts activities. An institutional climate 
supportive of community engagement and a democratic attitude toward knowledge production 
are also predictors of high quality broader impacts activities. The results also indicate that full 
professors are more likely to report high quality broader impacts activities, while researchers who 
submit to the NSF Directorate of Biological Sciences are less likely to do so. 

Table 5. Strength of Predictor Variables on Quality in Stepwise Regression 


Predictor Variable 

Beta 

n 

Expertise 

.312 

.000* 

Biology 

-.256 

.002* 

Democratic 

.247 

.003* 

Climate 

.236 

.005* 

Full professor 

.199 

.016* 


* Significant predictors at. 05. 


Open-ended Comments 

The survey concluded with an invitation for participants who wished to comment on the survey to 
do so in an open text box. Feedback from participants covered the following topics: appreciation 
for the study (4); suggestions to improve the study by clarifying the language (5) and allowing 
respondents to select multiple options (4); and complaints about the broader impacts criterion in 
terms of how the NSF manages it (2), how researchers respond to it (2), and the need for support 
(5). The strongest feedback focused on the need for support with broader impacts activities. This 
concern was expressed both in the comment box as well as through email responses to the author 
from sample members who chose not to complete the survey. 

Discussion 

Personal Characteristics 

All of the individual traits postulated in the literature to influence faculty community engagement 
examined in this study emerged, to varying degrees, as determinants of the broader impacts 
activities of academic faculty. 

Expertise. Broader impacts activities generally push investigators beyond their particular 
areas of specialization and require additional skill sets. Many scientists believe they do not 
have the skills necessary to develop appropriate outreach activities (Lok, 2010; Mathieu, 
Pfund, & Gillian-Daniel, 2009; Tretkoff, 2007). This theme pervading the literature was 
supported by the study’s finding that respondents had received little training to develop and 
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implement work similar to their broader impacts activities. Sample members in the survey and 
email responses to the researcher also reiterated this belief. 

The data analysis revealed that expertise was the strongest predictor of the type and quality 
of broader impacts activities. Because this variable emerged as such a significant predictor, 
the author repeated the analysis with a more conservative computation of faculty expertise 
by removing the items referring to prior experience and training as a graduate student. This 
narrower perspective on expertise did not diminish the significance of this variable in any of 
the regression analyses. The survey findings and supplemental responses underscore the need 
for faculty development and training in broader impacts, as well as institutional support to 
assist with implementation and evaluation. 

Epistemology. The influence of epistemology was manifested in this study in more than one 
way. Negative bias in the scientific community against the broader impacts criterion appears 
to have limited the survey response rate. Several members of the sample opted to reply to the 
author with negative comments rather than complete the survey. The following example of a 
response to the author’s email invitation to complete the survey provides anecdotal evidence 
of persistent resistance to the broader impacts criterion: 

“Most everyone I know that submits to NSF “HATES* to come up with arguments 
about broader impacts for NSF proposals. They hate doing it, they hate thinking 
they had to do it, etc. So, when a request like yours comes in, most folks can’t delete 
your unwelcome spam-like message fast enough.” 

The literature reviewed documents with similar negative responses from researchers, describing 
the broader impacts criterion as burdensome, counterproductive, and punitive (Tretkoff, 2007); 
confusing and frustrating (Sarewitz, 2011); irrelevant, ambiguous, and poorly worded (Holbrook 
& Frodeman, 2005); and undermining the autonomy of the scientific enterprise (Holbrook & 
Frodeman, 2011). 

Some of this resistance stems from the fact that the revisions to the merit criteria had little impetus 
from the scientific community, but were instituted in response to Congressional prodding (Bozeman 
& Boardman, 2009). The broader impacts criterion is interpreted by some as an introduction of 
extraneous political, cultural, or economic concerns into basic research (Holbrook & Frodeman, 
2007); bringing considerations external to science into the peer review process (Holbrook, 2005), 
and politicizing the value-neutral pursuit of science (Holbrook & Frodeman, 2007). Holbrook 
and Frodeman (2007) suggested that agitation over the broader impacts criterion is to be expected 
from anyone who believes that beneficial societal impacts will eventually follow from a laissez 
faire approach to basic research. Thus, reaction to the broader impacts criterion may depend on 
one’s conception of knowledge production, one’s views on the democratization of science, and the 
degree to which one supports the Bush model of opposition between basic and applied research. 

Survey responses revealed that epistemology shaped participants’ response to the broader impacts 
criterion, with a democratic orientation towards knowledge production emerging as a predictor 
of the quality of broader impacts activities. A democratic orientation recognizes that public 
engagement increases the acceptance of research results, generates information that meets user 
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needs, accounts for public values and knowledge, and improves the quality of science policy 
decisions. Participants shared an orientation towards knowledge production that was more 
democratic than technocratic. Tet, despite this appreciation for the socially embedded context 
of application, respondents favored research that was curiosity-driven rather than use-inspired. 
The apparent incongruity between respondents’ valuing of public engagement to increase the 
acceptance of results and meet user needs, while favoring research that is curiosity-driven rather 
than use-inspired, may indicate an emerging culture shift. Researchers appear to be embracing 
an alternative value system that acknowledges the need for public engagement while clinging to 
traditional norms regarding what constitutes proper academic science. 

A similar dynamic tension was evident in questions about how knowledge is generated. Survey 
respondents indicated a strong belief that knowledge is generated through trans-disciplinary 
collaboration that brings together multiple sources of distributed knowledge, a view characteristic 
of a democratic orientation. Interestingly, they also expressed the concurrent belief that knowledge 
is generated through objective disciplinary expert-led investigation, suggesting they did not 
perceive these opposing approaches to knowledge production as dichotomous. 

Rank. While prior research suggested a significant relationship between an investigator’s 
professional career status and engagement in community-based research, the direction of 
the relationship was inconclusive. It has been reported that the types of publicly engaged 
scholarship faculty members pursued varied by rank in statistically significant ways, with 
full professors more likely than associate professors to report such work (Glass, Doberneck, 
& Schweitzer, 2011); that lower ranked faculty were more frequently involved in public 
service (Antonio, Astin, & Cress, 2000); that faculty members of various ranks reported 
using scholarship to address community needs at similar levels (Vogelgesang, Denson, & 
Jayakumar, 2010); and that tenured and untenured faculty showed similar levels of frequency 
in conducting the scholarship of engagement (Braxton & Fuckey, 2010). Some argue that 
junior faculty trained in an environment where engaged scholarship is increasingly viewed as 
an important part of the mission of modern universities are more likely to engage than senior 
faculty who were not trained in such an environment (Jaeger & Thornton, 2006). Others 
contend that successful senior scholars with tenure have the flexibility and freedom to pursue 
non-traditional research (Scott, 2007). One finding that is consistently reported is that junior 
faculty members are routinely counseled to avoid service activity that may interfere with their 
research productivity (Stanton & Wagner, 2010). 

This study found that academic rank was a predictor of the quality of broader impacts activities. 
The finding that being a full professor was a significant predictor of quality aligns with reports 
in the literature that full professors are more likely than associate professors to conduct 
community engaged work (Glass, Doberneck, & Schweitzer, 2011) and that successful senior 
scholars with tenure have the flexibility and freedom to pursue non-traditional research 
(Scott, 2007). 

Discipline. The literature suggests that the ways in which faculty members define, design, 
and enact their community-engaged scholarship are largely driven by discipline (Doberneck, 
Glass, & Schweitzer, 2010). This aligns well with the study’s finding that discipline contributed 
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significantly to the predictive model for type of broader impacts activity. In this study, 
discipline was operationalized as the NSF directorate to which the researcher applied. The 
results revealed that researchers who submitted to the NSF Directorate ofBiological Sciences 
were less likely to report high quality broader impacts activities. This supports various reports 
in the literature that community engagement was less prevalent in the physical sciences 
(Lunsford & Omae, 2011; Vogelgesang, Denson, &Jayakumar, 2010). 

Institutional Support 

One of the key predictors of engaged scholarship is institutional practices and policies that support 
engagement (Lunsford, Bargerstock, & Greasley, 2010). Faculty motivation to engage in public 
scholarship is influenced by institutional characteristics including mission, resources, norms, 
and evaluation policies (Colbeck & Wharton-Michael, 2006). Some theorists have noted that 
institutional characteristics have a greater impact than personal motivation and commitment 
when it comes to scholarly productivity, and that faculty productivity may be stimulated or stifled 
by organizational culture and administrative structure (Freedenthal, Potter, & Grinstein-Weiss, 
2008). A study based on the Higher Education Research Institute’s 2004-2005 national survey of 
college and university faculty revealed that the perception of institutional support matters, even 
above and beyond the individual dispositions of faculty members, and even when disciplinary 
culture is accounted for. Every one-step increase in perceived institutional support was associated 
with a 10% increase in the likelihood of a faculty member collaborating with the local community 
in research or teaching (Vogelgesang, Denson, & Jayakumar 2010). Institutional support can be 
conveyed through both climate and resource allocation. 

Climate. Campus climate is a function of interpersonal interactions and includes dimensions 
such as “perceptions, attitudes, and expectations” (Cress, 2002, p. 390). A multiple regression 
analysis revealed that an institutional climate supportive of community engagement is a 
strong predictor of high quality broader impacts activities, accounting for 8% of the variance 
explained by the model. The literature identified the university reward system as the greatest 
obstacle to community engaged research (Pfirman, Collins, Lowes, & Michaels, 2005). 
Responses to the author’s survey concurred that promotion and tenure policies were the 
feature of institutional climate least supportive of community-based research. 

Resources. While faculty recognition and rewards are important motivators to initiate 
change, organizational structures must be in place to sustain the work (Brukardt, Holland, 
Percy, & Zimpher, 2004; Stanton, 2007). When the NSF asked the scientific community 
what role their institutions should play to ensure that broader impacts are realized, the 
majority response was that home institutions should increase support for broader impacts 
by establishing mechanisms for those activities, increasing financial or in-kind support, or 
encouraging involvement through changes in value systems and incentives (NSB, 2011). 
The allocation of resources for administrative support, faculty development, and incentives 
is the second largest barrier to engaged scholarship (Pfirman et al., 2005). Although resources 
emerged as a strong predictor of type ofbroader impacts activity, participants reported limited 
access to training or infrastructure to facilitate collaboration and assessment of community- 
engaged activities. 
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Type of Broader Impacts Activity 

NSF guidelines to proposers and reviewers identified general areas ofbroader impact: education, 
broadening participation, enhancement of infrastructure, broad dissemination, and other social 
benefit (NSF, 2008a). Despite admonitions from the NSF that Pis “are expected to go beyond their 
normal teaching duties and faculty commitments” (NSF, 2008b, p. 1), the category of education 
is by far the most common broader impacts activity, often with researchers simply repackaging 
what they are already doing (Lok, 2010). The chemistry division director at the NSF reported that 
“Overwhelmingly, the number one broader impacts that most people in the chemistry division 
are using is ‘training graduate students and postdocs’” (Lok, 2010, p. 417). In 2010, the National 
Science Board (NSB), the agency charged with NSF oversight, commissioned a topic modeling 
of the project summaries of approximately 150,000 NSF proposals. The analysis revealed that the 
education category occurred in more than 60% of the proposals (NSB, 2011). This was roughly 
three times the size of the next largest category. The results of the present study were congruent 
with the topic modeling; education was by far the most commonly reported activity at 57.3%. 

The America COMPETES Reauthorization Act of 2010 (P.L. 111-358), which provided 
reauthorization for the NSF, established new goals and policies for the broader impacts criterion. 
The Act stipulated that the NSF must apply this criterion to achieve an array of societal goals, 
including increasing economic competitiveness; developing a globally competitive workforce 
in science, technology, engineering, and math (STEM); expanding the participation of women 
and underrepresented minorities in science and engineering; increasing partnerships between 
academia, industry, and others; improving STEM education at PK-12 and undergraduate levels, 
including teacher development; increasing public scientific literacy; and expanding national 
security (Sec. 526[a][l]-[8]). The NSF integrated this expanded vision ofbroader impacts into a 
new Proposal and Award Policies and Procedures Guide (NSF, 2012). Congress’ identification of 
eight national goals to which broader impacts activities should be directed underscores the need 
for greater variation in the types ofbroader impacts activities Pis pursue. The results of this study 
suggest that limited resources and training constrain researchers to engaging in broader impacts 
activities that are the most familiar and least demanding (i.e. education). 

Quality of Broader Impacts Activity 

Hampering effective peer review ofbroader impacts is the fact that no standards have been set and 
no evaluation target provided (Bozeman & Boardman, 2009). The NSF does not systematically 
track how its broader impacts requirements are being met, nor does it have a system in place to 
evaluate project effectiveness (Lok, 2010). The first comprehensive analysis of the NSF’s review 
criteria, conducted in 2000, recommended that the NSF develop an evaluation strategy based 
on measures and performance indicators to track the objectives and implementation of the new 
criteria supported by both qualitative and statistical data collection methods capable of measuring 
incremental movement towards achieving the NSF’s strategic goals (National Academy of Public 
Administration, 2001). One of the six major themes of the second national review of the NSF 
criteria, conducted 10 years later, was that post-award assessment ofbroader impacts activities was 
weak and should be improved (NSB, 2011). Given the variation in size and scope of projects, the 
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report indicated that the effectiveness of broader impacts would be more meaningful if they were 
aggregated at a higher level than the individual project. 

Without established metrics and data collection, it is impossible to objectively assess the broader 
impacts of sponsored programs. Instead, this study asked Pis to evaluate the processes with which 
they devised and implemented their broader impacts activities using the standards of scholarship 
(clear goals, adequate preparation, appropriate methods, significant results, effective presentation, 
reflective critique, and ethical conduct). Given the subjectivity of self-assessment, better objective 
metrics of the quality of broader impacts activities are clearly needed. The criteria used in this 
study appeared to engender a social desirability bias. Respondents showed little variation in their 
high ratings of the quality of their broader impact activities, presumably because they wanted to 
believe that they were practicing good scholarship. It is likely that those who reported working 
with a specific community agency were more critical of these partnerships because such activity 
is traditionally less highly regarded in the academy and thus subject to a more frank assessment. 
Identifying valid and feasible ways to assess research impact more objectively should be a priority 
of future studies. Self-reported information needs to be balanced with multiple kinds and sources 
of data. An important contribution to the field would be the development of multiple well-defined 
measures, consisting of both hard and soft data, which can be systematically analyzed. 

Implications 

This study makes several contributions to the field of research administration. By providing a 
snapshot of the type and quality of broader impacts activities conducted by Pis, it enhances 
understanding of the training and support faculty members need to successfully meet the broader 
impacts requirements of federally sponsored research. In addition to data collection, the study 
examined the associations between relevant personal, professional, and institutional factors, 
and broader impacts activities, identifying patterns and highlighting barriers and facilitators to 
guide the decision making of higher education administration as well as federal funding agencies 
seeking to expand or enhance the broader impacts of university research. This study was designed 
to stimulate discussion among higher education administrators about community engagement, 
inform initiatives to expand interactions between campuses and external entities, generate strategies 
to improve the quality of broader impacts activities, and identify faculty development objectives 
that are scholarship-based. Linking broader impacts activities with forms of engaged scholarship 
reinforces the value of such work and establishes new indicators for the institutionalization of 
engagement at research universities. 

The results of the study also hold practical implications. Expertise emerged as the strongest 
predictor of broader impacts activity type and quality, yet respondents reported receiving little 
training in such work. The study identified the need for faculty professional development in 
effective evaluation, putting the results to use, establishing clearly defined partnerships with 
community agencies, and equitably distributing power and responsibility with community 
partners. Such training need not be limited to faculty. Respondents reported that they had 
received little to no training and experience in community engagement as graduate students. 
Because graduate school is where researchers are socialized as scholars and where they develop 
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their professional identities, extending training to graduate students and postdoctoral researchers 
will help create a cadre of professionals who conduct high quality broader impacts activities as a 
natural part of their research programs. 

While there is clearly a need for training, faculty may be reluctant to pursue it. This disinclination 
was expressed both via email responses and through the surveys open comment box. Participants 
asserted that they lack both the time and desire to learn how to implement broader impacts 
activities; that it did not make sense to require scientists to do such work; that trained experts are 
better suited to public outreach than research scientists; and that it would be preferable to partner 
with experts in broader impacts than to learn how to conduct such activities. These comments 
point to the need for campuses to provide professional staff trained in implementing broader 
impacts activities to collaborate with and support researchers. 

Survey respondents reported limited infrastructure available to assist with their broader impacts 
work. Existing service-learning and civic engagement centers, or other campus-wide coordinating 
units, can fill that gap by providing researchers with practical assistance; links to community 
partners; information about established, well-aligned projects with which to connect; and a venue 
for interdisciplinary partnerships. Technical assistance would be especially important in assessment 
and evaluation of the outcomes of broader impacts activities. 


Beyond investing in training and infrastructure, institutions can also enhance and expand the 
quality and type of their broader impacts activities by providing incentives (such as stipends, 
release time, and seed grants) for faculty to build engagement into their work. An institutional 
climate supportive of community engagement was a strong predictor of high quality broader 
impacts activities. An institutions climate is shaped by values expressed through rhetoric, public 
recognition, and reward systems. Respondents indicated that the executive leadership at their 
institutions explicitly promoted community engagement as a priority, but their promotion and 
tenure policies did not recognize anything other than peer-reviewed publications and grants 
as measures of productivity. If researchers are to be motivated to conduct high quality work in 
the community, tenure and promotion guidelines must be revised to ensure that they receive 
professional recognition and advancement for engaged scholarship. 


Overhauling the faculty-reward system will not work without a broad shift in the attitudes of 
professors, especially those on promotion and tenure committees. Simply adding new metrics 
or categories of work to evaluate as part of the faculty reward systems is not enough. Academics 
must be encouraged to think differently. Executive leaders can inculcate a supportive climate by 
publicly promoting community engagement as a priority and ensuring that engagement activity 
is publicly recognized through award programs, university publications, and events that showcase 
the work of engaged scholars. By creating new traditions, rituals, and symbols that reinforce the 
value of community-based work, institutions can foster an epistemological shift toward democratic 
engagement and ease the dynamic tension accompanying the transition from traditional to 
alternative value systems, creating enduring and pervasive change. 


With decreasing appropriations for public higher education and shrinking budgets, academics 
are under growing pressure to secure external funding for their research. In response to prodding 
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from Congress, the NSF and other federal funding agencies are placing more, not less, emphasis on 
public accountability for and engagement in federally sponsored research. As prestigious federal 
grants increasingly require an outward-looking orientation, more and more scientists are finding 
themselves compelled to practice engaged scholarship. Research administrators must continue to 
explore how investigators can be both socialized and supported so that they embrace community 
engagement as an important and necessary element of federally funded research. 
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